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Abstract 


The public, education, and workforce sectors all have expressed interest regarding the key 
knowledge, skills, and abilities that enable individuals to be productive members of society. 
Although past efforts have attempted to create frameworks of student learning outcomes, the 
results have varied due to different perspectives and goals. Thus, the purpose of this paper was to 
gather and review relevant higher education frameworks, detennine their commonalities and 
create domains, and identify the assessments that Educational Testing Service (ETS) has 
developed in each of the domains. After a thorough review of the relevant frameworks, seven 
key domains were identified: creativity, critical thinking, teamwork, communication, digital and 
information literacy, citizenship, and life skills. Also discussed were the issues of education 
versus work contextualization and the assumption of foundational quantitative reasoning and 
literacy skills infonning these seven domains. 

Key words: student learning outcomes, higher education standards, higher education, creativity, 
critical thinking, teamwork, communication, digital and information literacy, citizenship, life 
skills 
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In 2006, the Commission on the Future of Higher Education, under the guidance of then 
Secretary of Education Margaret Spellings, released A Test of Leadership: Charting the Future 
ofU.S. Higher Education (U.S. Department of Education, 2006). The report painted the picture 
of a changing landscape where colleges and universities increasingly face international 
competition; low rates of access and degree attainment; and questions about the knowledge, 
skills, and ability of their graduates. Among other recommendations, this report called for 
institutions to demonstrate better what students have learned and can do as a result of their 
college experience. 

Another source of pressure comes from within higher education. In 2006, the National 
Association of State Universities and Land-Grant Colleges and the American Association of 
State Colleges and Universities published the Toward a Voluntary System of Accountability 
Program (VSA) for Public Universities and Colleges (McPherson & Shulenburger, 2006). The 
Voluntary System of Accountability (VSA) was reportedly designed to increase the quality of 
higher education, both to be more transparent for its constituents, but also as a responsibility 
accordant with its vital role in American societal and economic success. The VSA provided a 
framework of methods and assessments to gather data across institutions in three core 
educational outcomes: critical thinking, analytical reasoning, and written communication skills. 

Although some have argued that the VSA was merely a response to calls from the 
Spellings Report (even though the project was underway before the commission’s; Miller, 2008), 
other efforts to define and measure key areas of student learning have come from within 
academe. For example, the Association of American Colleges and Universities (AAC&U) also 
began the Liberal Education and America’s Promise (LEAP) initiative in 2005. Seeking to 
“recalibrate college learning to the needs of the new global century” (AAC&U, 2007, p. vii), 
LEAP focuses on defining key student learning outcomes, disseminating best practices in 
instruction, and encouraging quality assessment among its members. For institutions of higher 
education and the organizations that represent them, measuring student learning serves the 
function of ensuring quality, both internally and to their constituents. 

Finally, calls for better evidence of student learning have come from employers. Several 
recent reports have stated that, despite high rates of unemployment, employers claim they cannot 
find sufficiently skilled workers (Deloitte & The Manufacturing Institute, 2011; 
ManpowerGroup, 2012). This skills gap is seemingly a result of employer dissatisfaction with 
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the products of higher education. A study by the Conference Board, Corporate Voices for 
Working Families, Partnership for 21st Century Skills, and the Society for Human Resource 
Management surveyed more than 400 employers across the United States, asking their 
perceptions of those currently entering the workforce (Casner-Lotto & Barrington, 2006). The 
survey did not focus exclusively on college graduates but referred questions to those entering the 
workforce with high school diplomas, 2-year degrees, and 4-year degrees. Respondents 
emphasized the need for more applied skills, such as oral and written communication, teamwork, 
and professionalism, over basic academic knowledge in areas such as English, math, and science. 
Additionally, over 25% of respondents indicated that college graduates were deficient in several 
key skills, such as leadership and written communication. 

A study conducted for the AAC&U by Hart Research Associates (2010) sampled over 
300 executives from a wide array of organizations to detennine their perceptions of the needs of 
college graduates. Of those surveyed, 88% said that graduates need higher levels of learning and 
skills than they had in the past. Only 28% of respondents agreed that 4-year colleges and 
universities are doing a good job of preparing students for the modern workforce. Thus, the 
workforce sector is vested in defining and measuring student learning so that it can ensure a 
sufficiently skilled supply of new employees. 

These three sectors—public, education, and workforce—all pose different questions to 
higher education as they weigh the high cost of a degree. Are students acquiring the key 
knowledge, skills, and abilities to function as productive members of society? Are college 
graduates ready to enter and succeed in the workforce? Though differing in origin, each of these 
questions has led members of these respective sectors to discuss the focus of higher education, 
resulting in a litany of proposed frameworks for student learning. 

Literature that seeks to define student learning in higher education comes from diverse 
perspectives and targets diverse goals. Organizations in education, government, public policy, 
and industry all have a vested interest in the direction of higher education. These interests vary 
from societal good to human capital development to self-detennination and autonomy. These 
frameworks take many forms. Some, such as the VS A (McPherson & Shulenburger, 2006), 
simply mention key areas, such as broad general education outcomes, on which higher education 
should focus. Others, such as the Conference Board study (Casner-Lotto & Barrington, 2006), 
conduct surveys of key constituents to detennine what factors are important to them. 
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As a result of varying motives and methods, research in this area has come to a range of 
conclusions. Should higher education focus on the traditional liberal arts education or gear 
toward work-relevant competencies? Should the framework for higher education be broad, 
leaving room for local determination, or specific as to ensure continuity across graduates? These 
are just some of the questions that arise. Ultimately, this multitude of efforts to create common 
understanding across higher education has led to some confusion. 

Purpose 

We began this project with three goals: 

1. To gather and review outcomes frameworks relevant to higher education, 
considering the social, educational, and occupational perspectives. This focus on 
higher education is a key consideration of this paper. Though preparing students for 
the workforce is and always will be an important role of many educational systems, a 
framework of outcomes designed to define what a student should leam and have 
achieved in college is different than one that seeks to certify the competencies 
necessary for entering the workforce. 

2. Determine commonalities among these frameworks. Our team thoroughly reviewed 
the relevant literature to identify and define common areas of learning. Our review 
yielded seven key domains that largely represent these established frameworks. 

3. Identify assessments that Educational Testing Service (ETS) has developed in each of 
these domains and the extent to which the assessments align with the definitions 
presented here. By consulting with representatives from across ETS, we have 
identified both operational assessments and research tools that have been designed to 
assess many, but not all, of these domains. This alignment will be discussed in the 
final section of the report. 


Method 

Selection Criteria 

Using several relevant frameworks, the research team assembled them in two ways. First, 
we looked at the domains—broad, high-level skills—within each framework. By comparing and 
contrasting these across all domains and determining those factors that were included in a 
majority (at least four of seven) of the frameworks reviewed, we were able to identify the seven 
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critical domains presented below. Next, we examined the operational definitions—the learning 
outcomes used to define each domain. By analyzing these, we were able to identify the 
knowledge, skills, and abilities that define the critical domains. 

Given the three goals stated above, we used three criteria to select relevant frameworks: 
population criterion, scope criterion, and specificity criterion. For the purpose of this paper, the 
population criterion referred specifically to frameworks targeted at higher education. For the 
scope criterion, we focused on frameworks that span across disciplines and are appropriate for all 
institutions of higher education. Finally, the specificity criterion included only those frameworks 
that provided operational definitions of the knowledge, skills, and abilities discussed in the 
framework. A more thorough discussion of each criterion is below. 

Frameworks targeted toward higher education (population criterion). Many 
previous efforts to identify key skills have focused on either basic work requisite skills or general 
education outcomes. For example, the National Work Readiness Certificate was developed using 
work-contextualized cognitive test items representing the ability to use applied mathematics, 
applied reading, and information location as part of workplace task performance. It is widely 
used within the community college and career technical training sectors as it is most appropriate 
and specific to work training readiness and workplace success. From another perspective, several 
educationally focused frameworks have been created to understand higher education readiness 
and success by either focusing on primary and secondary education (e.g., the Common Core 
State Standards) or spanning all levels of education (e.g., National Research Council [NRC], 
2008,2010, 2011,2012). 

Much of the debate about the value of postsecondary education, as well as questions of 
degree quality, has focused on 4-year (i.e., bachelor’s) degrees. Thus, we only considered 
frameworks that explicitly targeted a bachelor’s level education or, in the case of workforce 
examples, the expected skills of those holding a 4-year degree. 

Multidisciplinary and trans-institutional (scope criterion). Given the emphasis on 
assessment and accountability over the past several decades, many individual institutions of 
higher education have developed frameworks of learning outcomes for their students. Similarly, 
professional organizations and fields vary in the requirements and expectations of their members. 
Thus, we considered only educational frameworks that were intended to go beyond any one 
institution or field of expertise. For example, a recent study by Shultz and Zedeck (2012) 
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outlined characteristics that were indicative of effectiveness in the law profession. Similarly, the 
U.S. Department of Labor, Employment and Training Administration (USDOL-ETA, n.d.) 
published a competency model specifically for advanced manufacturing. Although these 
frameworks may provide valuable insight for some key skills, we chose to focus on those that 
were identified as vital across fields. 

Specific in the learning outcomes or competencies expected (specificity criterion). 

Many efforts that outline key skills for learners and workers stop short of providing operational 
definitions of these skills. In reviewing the literature, we saw this embodied in several ways. 
Efforts that commonly use the tenn 21st century skills (e.g., NRC, 2008, 2010, 2011, 2012) have 
thoroughly reviewed the scientific literature and often refer to skills such as critical thinking, 
communication, and teamwork, among others that we discuss here. Similarly, surveys of 
employers (e.g., AAC&U, 2010; Casner-Lotto & Barrington, 2006; Hart Research Associates, 
2010; Manyika et al., 2011) broadly refer to these same skills. In both cases, however, 
operational definitions that include supporting learning outcomes or competencies—statements 
about specific knowledge, abilities, and dispositions inherent to these domains—were lacking. 
For the purposes of this report, such specificity was necessary to generate a common 
understanding among all groups, as well as to inform potential future work in curriculum and 
assessment development. 

Frameworks Reviewed 

Given these criteria, we identified seven frameworks for consideration (see Table 1). We 
had considered organizing these frameworks according to the sector of the publishing 
organization (i.e., policy/govemment, education, workforce), but the work itself is rarely as 
discrete as such segmentation would suggest. Though many of the published reports around these 
efforts come from a single entity, the frameworks themselves are often vetted, if not developed 
in collaboration with, members of other sectors. The AAC&U’s LEAP framework was 
developed with significant input from employers. The Assessment & Teaching of 21st Century 
Skills (ATC21S) framework was initiated by a consortium of corporate partners—Cisco, Intel, 
and Microsoft—but conducted in collaboration with the University of Melbourne and other 
institutions of higher education. 
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Table 1 


Frameworks of Learning Outcomes 


Framework 

Abbreviated title 

Author/impetus 

Framework for Higher Education 
Qualifications 

QAA-FHEQ 

Quality Assurance Agency for 

Higher Education 

European Higher Education Area 
Competencies 

Bologna 

European Commission: European 
Higher Education Area 

Liberal Education and America’s 
Promise 

LEAP 

Association of American Colleges 
and Universities 

Frameworks for Learning and 
Development Outcomes 

CAS 

The Council for the Advancement of 
Standards in Higher Education 

The Degree Qualifications 

Profile 

DQP 

The Lumina Foundation 

The Assessment & Teaching of 
21st Century Skills 

ATC21S 

Collaboration among Cisco, Intel 
Microsoft, the University of 
Melbourne, and others 

ETA Competency Model 
Clearinghouse’s General 
Competency Model Framework 

USDOL-ETA 

U.S. Department of Labor, 
Employment and Training 
Administration 


With that said, many of the frameworks are written to meet the goal of the sponsoring 
organization. For example, the framework put forth by USDOL-ETA, though informative for 
higher education, is designed to represent the skills of employees and not the knowledge, skills, 
abilities, and mindset of a well-rounded citizen (a term frequently used by Lumina’s Degree 
Qualifications Profile, LEAP, and others). Similarly, the Degree Qualifications Profile (DQP) 
outlines the abilities of degree holders, not necessarily prospective employees. However, as we 
ultimately seek to synthesize these models, these different perspectives will strengthen the final 
product, as the domains that are common across all three sectors will have been determined with 
broader endorsement. 

Quality Assurance Agency for Higher Education—Framework for Higher 
Education Qualifications (QAA-FHEQ). In 1997, the National Committee of Inquiry into 
Higher Education released their Dearing Report (see Dearing, 1997), which indicated a great 
need to create a national framework for higher education qualifications (Quality Assurance 
Agency [QAA], 2008). In response, the Framework for Higher Education Qualifications (FHEQ) 
in England, Wales, and Northern Ireland was developed by the QAA (2008). In 2001, the FHEQ 
published the first edition of the framework and completed an updated second edition in 2008. 
This framework has been designed to meet the standards of the Bologna Declaration on the 
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European Space for Higher Education as well as align with the Framework for Qualifications of 
the European Higher Education Area (QAA, 2008). 

Briefly, the purposes of the FHEQ, as described in The Framework for Higher Education 
Qualifications in England, Wales and Northern Ireland: August 2008 (QAA, 2008), are the 
following: 

• to provide important points of reference for setting and assessing academic standards 
to higher education providers and their external examiners, 

• to assist in the identification of potential progression routes, particularly in the 
context of lifelong learning, and 

• to promote a shared and common understanding of the expectations associated with 
typical qualifications by facilitating a consistent use of qualifications titles across the 
higher education sector (QAA, 2008, p. 6) 

The FHEQ is a “qualifications framework” (QAA, 2008, p. 7) with five levels. These 
levels are numbered 4-8 (following Levels 1-3 as indicated by the National Qualifications 
Framework and the Qualifications and Credit Framework), with Level 4 referencing higher 
education certificates, Level 5 indicating foundation degrees and/or diplomas of higher 
education, Level 6 including bachelor’s degrees, Level 7 referring to master’s degrees, and the 
highest FHEQ level, Level 8, including doctoral degrees (QAA, 2008). Each level has specific 
qualifications and skills associated with it. A qualification descriptor provides a statement of 
specific outcomes an individual should demonstrate as well as a global description of the abilities 
one should possess at that level (QAA, 2008). 

European Higher Education Area Competencies—The Bologna Declaration and the 
Tuning Project. In 1999, the Bologna Declaration was signed by higher education officials from 
29 European countries in an effort to create a greater level of trust and transparency throughout 
the European education systems (Education, Audiovisual, and Culture Executive Agency 
[EACEA], 2012). Thus began multiple proceedings, both formally and informally, to assist with 
these reforms, the most notable of which was the Tuning Project. Funded by the European 
Commission (the executive body of the European Union), the Tuning Project was developed 
through the efforts of faculty members (Adelman, 2009). The term tuning refers to the 
development of a framework that ascertains specific learning expectations for students in each 
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area of study. In turn, clear expectations are set for each subject area while academic autonomy 
is retained (Marshall, Kalina, & Dane, 2010). 

While the initial Bologna Declaration sought to create a system based on two cycles— 
undergraduate and graduate—and to specify requirements to transition from the first to the 
second (EACEA, 2012), the Tuning Project assisted with the design, development, 
implementation, and evaluation of each cycle (Tuning Educational Structures in Europe, n.d.). 
Thus, while similar, the Bologna Declaration and Tuning Project had different objectives. The 
Bologna Declaration led to the development of the National Qualifications Framework, which 
attempts to articulate the differences between qualifications in each cycle (EACEA, 2012). In the 
Tuning Project, the goal is to provide qualifications on the institutional and pan-European field 
levels. The Tuning Project provides faculty members and institutions with the information to 
adequately describe “cycle degree programs at the level of subject areas” (e.g., biology, 
sociology; Adelman, 2009, p. 48). 

To do this, faculty members are assigned the task of developing reference points and 
statements of learning outcomes, levels of learning, and desired competencies in particular 
disciplines using a methodology that ensures all statements are transparent and comparable 
(Adelman, 2009). The Tuning Project asserts that unifonnity is not ideal in degree programs, but 
rather the ability to find points of convergence, reference, and common understanding between 
them. Tuning articulates what a specific curriculum/program intends to do, but does not attempt 
to map one specific pathway (Adelman, 2009; Gonzalez & Wagenaar, 2003). 

Association of American Colleges and Universities—Liberal Education and 
America’s Promise (LEAP). In 2005, LEAP was established and has since served as an 
advocate for liberal education. Working with hundreds of higher education institutions at all 
levels (i.e., community colleges, colleges, and universities), LEAP provides information about 
“key outcomes of a quality education” (AAC&U, 2011, p. 1) and indicates how well institutions 
are assisting students in reaching the desired outcome through research and reports (AAC&U, 
2011). In addition to this research, LEAP has established several partnerships with colleges and 
universities to create the Campus Action Network (e.g., individuals from campuses across the 
country work together to discuss best practices in the field) and has formed the National 
Leadership Council (composed of business persons, academics, and policy leaders who advocate 
for all students to have access to opportunities in key education outcomes; AAC&U, 2011). 
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Central to the LEAP initiative are the views set forth in College Learning for the New 
Global Century issued by the LEAP National Leadership Council in 2007 (AAC&U, 2011). The 
report argued the importance of focusing on key education and learning outcomes and offered 
recommendations for achieving these results (AAC&U, 2011). Using the report and research as a 
guide, LEAP has denoted four domains that are considered “the essential learning outcomes”: 
knowledge of human cultures and the physical and natural world, intellectual and practical skills, 
personal and social responsibility, and integrative and adaptive learning (AAC&U, 2011, p. 7). 

In addition to the Essential Learning Outcomes framework, the LEAP National 
Leadership Council has proposed a new framework for excellence that includes seven principles: 

1. Aim high—and make excellence inclusive 

2. Give students a compass 

3. Teach the arts of inquiry and innovation 

4. Engage the big questions 

5. Connect knowledge with choices and action 

6. Foster civic, intercultural, and ethical learning 

7. Assess students’ ability to apply learning to complex problems (AAC&U, 2011) 

LEAP has emphasized the importance of developing a partnership between educators, students, 
and society to reach this level of excellence (AAC&U, 2011). 

The Council for the Advancement of Standards in Higher Education (CAS). In 1979, 
the Council for the Advancement of Standards in Higher Education (CAS) began with the 
purpose of creating, developing, and disseminating standards in higher education to educators 
and their institutions (Strayhorn, 2006). Specifically, CAS aims to support and enhance the 
quality of programs and services in the field of learning and development (CAS, n.d.). In an 
effort to meet these goals, delegates from 36 professional higher education associations, 
represented by a range of organizations in the United States and Canada—including both for- 
and not-for-profit organizations—serve as members of the CAS Board of Directors (CAS, n.d.). 

CAS has developed standards of professional practice for a range of areas in the realm of 
education and has created 16 learning and development domains, including intellectual growth, 
effective communication, realistic self-appraisal, enhanced self-esteem, clarified values, career 
choices, leadership development, healthy behaviors, meaningful interpersonal relationships, 
independence, collaboration, social responsibility, satisfying and productive life styles, 


9 



appreciation of diversity, spiritual awareness, and achievement of personal and educational goals 
(Strayhom, 2006). These domains are used to organize CAS’s frameworks and are further 
described in the report, Frameworks for Assessing Learning and Development Outcomes 
(Strayhom, 2006). Each framework is designed to help educators and practitioners assess 
students and evaluate procedures in a particular domain (e.g., intellectual growth) that are 
centered on both student learning and development (Strayhom, 2006). For each domain, the 
framework includes an introduction, theoretical context, related variables and indicators, 
examples of both quantitative and qualitative methods, tools available in the public domain, and 
related materials, such as websites, references, and readings (Strayhom, 2006, p. 12). 

The Lumina Foundation—Degree Qualifications Profile (DQP). In 2000, the 
privately funded Lumina Foundation for Education began its mission to enroll and graduate more 
students from college, with a specific focus on increasing the enrollment and graduation rates of 
four groups of students: low income, first generation, persons of color, and adult learners 
(Lumina Foundation, 2012). By 2025, the foundation has set a goal to increase the proportion of 
Americans who hold high-quality degrees and credentials to 60% (Adelman, Ewell, Gaston, & 
Schneider, 2011). Toward this end, the Lumina Foundation developed the DQP. 

The DQP provides a framework to describe what students, at any educational level, 
should know and have the ability to do with a particular degree (Adelman et ah, 2011). Levels of 
achievement are defined for associates’, bachelor’s, and master’s degrees. Thus, the profile 
demonstrates how students should perform at increasingly challenging levels (Adelman et ah, 
2011). Regardless of one’s field of study, the DQP provides descriptions for five areas of 
learning: broad, integrative knowledge; specialized knowledge; intellectual skills; applied 
learning, and civic learning (Adelman et ah, 2011). The descriptors and outcomes are the end 
result of a review of over a decade of research across all levels of U.S. higher education to define 
the expected learning outcomes that are necessary for success at work, citizenship, and global 
participation (Adelman et ah, 2011). The Lumina Foundation has noted that in practice these 
areas overlap and should be integrated. 

The Assessment & Teaching of 21st Century Skills (ATC21S). In 2009, Cisco, Intel, 
and Microsoft officially sponsored the international research project ATC21S, led by the 
University of Melbourne in Australia and in cooperation with the countries of Australia, Finland, 
Singapore, and the United States (ATC21S, 2012a). The goal of this partnership is to prepare 
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students for the 21st century workforce by enhancing educational systems with the incorporation 
of skills such as collaboration and digital literacy in curricula through assessment. 

The ACT21S reports that few nations describe their national curriculum in detail, but 
many have set high goals for their educational system, often mentioning 21st century skills but 
rarely providing more infonnation than a reference to the skills (Binkley, Erstad, Hennan, 

Raizen, & Ripley, 2009). Thus, ATC21S developed a five phase plan, ranging from defining the 
current state of education to creating public domain resources to build a foundation that will 
enable 21st century skills to resonate with all students throughout the world (ATC21S, 2012b). 

Phase 1 of the five phase plan is relevant to the purpose of this paper. During this phase 
the project’s five working groups reviewed the existing literature and frameworks and produced 
white papers that have informed the development of a new framework to guide the creation of 
large-scale assessments of 21st century skills. The outcome of this synthesis is the KSAVE 
framework. KSAVE refers to the knowledge, skills, attitudes, values, and ethics associated with 
the important skills, which KSAVE identifies for success in the 21st century (Binkley et ah, 
2009). The KSAVE framework uses 10 skills, grouped into four categories. The categories and 
skills (skills provided in parentheses) include: ways of thinking (creativity and innovation, 
critical thinking, problem solving, decision making, learning to leam, metacognition), ways of 
working (communication, collaboration [teamwork]), tools for working (information literacy, 
information and communication technologies [ICT)] literacy), and living in the world 
(citizenship—local and global, life and career, personal and social responsibility). For each of 
these skills, the KSAVE framework provides reference to the knowledge requirements, the 
abilities and skills associated that guide curriculum, as well as the behaviors and aptitudes related 
to a particular skill (Binkley et ah, 2009). 

United States Department of Labor, Employment and Training Administration 
(USDOL-ETA). In the last few decades, the USDOL-ETA has recognized a clear need to focus 
on and develop a competency model to assist in the determination of the necessary requirements 
for workers, based on the needs of employers and businesses (Ennis, 2008). To meet this end, the 
USDOL-ETA developed the ETA Competency Model Clearinghouse’s General Competency 
Model Framework. The competencies included in this framework are represented by nine tiers, 
organized in a hierarchical, or stackable, fashion, with general competencies at the bottom and 
job-specific competencies at the top The arrangement of the competencies from bottom to top are 
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as follows: personal effectiveness competences, academic competencies, workplace 
competencies, industry-wide technical competencies, industry-sector technical competencies, 
occupation-specific knowledge areas, occupation-specific technical competencies, occupation- 
specific requirements, and management competencies. 

The tiers at the lower levels (i.e., personal effectiveness competencies, academic 
competencies, workplace competencies) represent foundation competencies. These foundational 
competencies are broad competencies, such as initiative (personal effectiveness competencies), 
mathematics (academic competencies), and teamwork (workplace competencies) that can be 
applied across many industries (Ennis, 2008). Tier 4 (industry-wide technical competencies) and 
Tier 5 (industry-sector technical competencies) refer to competencies specified by a particular 
sector or representative of an industry. The final group of the USDOL-ETA framework includes 
Tiers 6-9 and focuses on competencies and knowledge specific to an occupation (Ennis, 2008). 

Results: The Critical Domains 

As a result of reviewing these frameworks, we identified seven common critical domains. 
Each domain is presented and described below, including supporting language from the source 
frameworks. See Table 2 for a concise summary of these results. 

Creativity 

The ability to generate new ideas is an important competency for postsecondary students 
to exhibit. At the same time, it is also essential for students to apply these ideas and solutions in a 
real-world context. Taken together, creative thinking is presented here. Indeed, these concepts 
were discussed broadly in the literature that was reviewed, with five of the seven frameworks 
addressing similar concepts. When considering all of the frameworks reviewed, creative thinking 
can be defined as 

• generation of new ideas, 

• novel integration of existing ideas, and 

• application of new ideas in a real-world setting. 

First, students must be able to generate new ideas. This commonly held definition of 
creativity was emphasized by several of the frameworks reviewed here. In defining its “creative 
thinking” outcome, LEAP includes “thinking, reacting, and working in an imaginative way” 
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(Rhodes, 2010, p. 3). Idea generation is also directly referenced in outcomes in both the ATC21S 
and Bologna frameworks. 

Table 2 

A Summary of the Seven Critical Domains and Use in Educational Testing Service (ETS) 


Products and Assessments 


Domain 

Components 

Domain measured in ETS products 




TOEFL/ 




EPP Criterion „ iSkills 

I UmC 

PPI b 


The generation of new ideas 



Creativity 

Novel integration of existing ideas 
Application of new ideas in a real- 
world setting 

Thinking critically 


X 

Critical 

Solving problems 

X 


thinking 

Synthesize information 

Sense-making 

Fulfdl roles within a team 




Teamwork 

Treat group members with respect 
Motivate group members 

Effectively communicate multiple 


X 

Effective 

types of messages 



communica¬ 

Communicate across multiple forms 

XX X 

X 

tion 

Effectively deliver messages to 
varying audiences 

Accessing and finding information 



Digital & 

Analyzing and evaluating 



information 

information 

X 


literacy 

Using and managing information 
Applying technology effectively 

Civic knowledge 

Participating in civic processes 



Citizenship 

Action and organization toward 
change 

Respect for others 

Ethics and integrity 

Independence, self-directed 
learning 



Life skills 

Time management 

Goal setting 

Adaptation, flexibility 


X 

Note. EPP = 

ETSt® Proficiency Profile, PPI = ETSf® 

Personal Potential Index. 


a The TOEFL® and TOElC® programs assess communication skills for similar populations, but 

these assessments deal specifically with assessing English learners. b Issues of methodology, 
audience, use, and intended population must be considered when using the PPI. 
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Second, creativity does not simply refer to idea generation, but the novel integration of 
existing ideas. One criterion of the Lumina DQP stated that students should produce “creative or 
practical work that draws on specific theories, tools and methods from at least two academic 
fields” (Adehnan et ah, 2011, p. 11). Similarly, LEAP refers to “the capacity to combine or 
synthesize existing ideas” (Rhodes, 2010, p. 3). 

Third, creativity includes the application of ideas in a real-world setting. LEAP 
recommends frequent and extensive practice of creative thinking “across the curriculum, in the 
context of progressively more challenging problems, projects and standards of practice” 
(AAC&U, 2007, p. 3). 

Critical Thinking 

Critical thinking, problem solving, and decision making are important features of the 
primary, secondary, and postsecondary curriculum in many parts of the world as evidenced by 
the attention given to these competencies on large-scale assessments (e.g., National Assessment 
of Educational Progress [NAEP] and Program for International Student Assessment [PISA]). 
Taken together, critical thinking, problem solving, and decision making incorporate one’s ability 
to reason effectively, use systems thinking and evaluate evidence, solve problems, and clearly 
articulate the result of the inquiry and exhibit. There is a great deal of consistency when looking 
across frameworks for the inclusion of critical thinking, problem solving, and decision making as 
all seven of the frameworks addressed similar concepts. When considering all of the frameworks 
reviewed, critical thinking, problem solving, and decision making can be defined as 

• thinking critically, 

• solving problems, 

• synthesizing information, and 

• sense-making. 

Thinking critically and synthesis of infonnation refers to one’s ability to understand the 
importance of evidence, interpret infonnation, and draw conclusions based on the best analysis 
(Bologna, CAS, Lumina DQP, QAA-FHEQ, USDOL-ETA). Second, problem solving and 
decision making refers to one’s ability to ask questions that clarify various points of view and 
lead to better solutions (Bologna, CAS, Lumina DQP, QAA-FHEQ, USDOL-ETA). Finally, 
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sense-making refers to one’s ability to determine the deeper meaning or significance of what is 
being expressed (QAA-FHEQ, CAS). 

In addition to agreement across frameworks on the inclusion of core descriptors for 
critical thinking, problem solving, and decision making, some frameworks expand on these 
definitions. For example, the ATC21S also emphasizes the importance of an attitudinal 
disposition, which includes being trustful of reason, open and fair in deed, flexible, and honest, 
as well as inquisitive and well infonned. 

Teamwork 

Our review of the frameworks clearly conveyed the prevalence of team-based work in 
both academic and professional settings. To this end, teamwork and collaboration have become 
crucial domains for higher education. However, explicitly defining the necessary facets of 
collaboration could be the focus of entire fields of study. Thus, it was no surprise that various 
frameworks discussed this domain have great variance in manner and depth. Nevertheless, five 
of the seven frameworks reviewed referenced the need for teamwork, with the common tenants 
across them including the following: 

• Fulfill roles within a team. 

• Treat group members with respect. 

• Motivate group members. 

• Possess leadership skills. 

Although several frameworks refer broadly to the ability to function as a productive team 
member (Bologna, CAS), key behaviors emerge from more extensive definitions. Several models 
(e.g., CAS, LEAP, USDOL-ETA) refer to effective communication, including receiving and 
providing feedback as part of one’s role within a team. Students must also uphold obligations, 
completing tasks that are assigned to them (CAS, LEAP, USDOL-ETA). 

Second, students must treat other members of groups with respect. When conflict arises, 
students must be adept at managing those situations (Bologna, CAS, USDOL-ETA). They 
should also appreciate both demographic and experiential diversity in their teams (ATC21S, 
Bologna). 

Third, five of the seven frameworks reviewed mentioned the need for students to be able 
to motivate or inspire action in others. LEAP suggested that students should motivate teammates 


15 



by expressing confidence about the importance of the task and the team’s ability to accomplish 
that goal (AAC&U, 2011). According to the Bologna competencies, students must have the 
“ability to motivate people and move toward common goals” (Lokhoff et ah, 2010, p. 63). 

Finally, we note that leadership skills were mentioned in four of the seven frameworks 
reviewed. Just as with teamwork, leadership is a well-studied and often debated topic. However, 
one might intuit that leadership is a vital outcome for some or even many, but not all, graduates 
of higher education. 

Effective Communication 

All seven reviewed frameworks devoted significant attention to communication in its 
various forms. In the aforementioned Conference Board Survey (Casner-Lotto & Barrington, 
2006), both oral and written communication were among the most needed skills mentioned by 
employers saying that these competencies (95% and 93%, respectively) were “very important.” 

Communication can be defined as a student’s ability to 

• effectively communicate multiple types of messages, 

• communicate across multiple fonns, and 

• effectively deliver messages to varying audiences. 

Across more than 80 individual outcomes that referred to communication, one unique 
core tenant was the importance of quality communication. At times this was mentioned very 
directly: “writes coherently” (CAS, Strayhorn, 2006, p. 46), “speaks effectively” (CAS, 
Strayhom, 2006, p. 46), and clearly expresses the findings of inquiry (ATC21S, Binkley et ah, 
2009). Other frameworks were more explicit in defining quality communication. The USDOL- 
ETA framework outlined several fundamentals of proper writing mechanics and speaking. 
Similarly, LEAP provided entire rubrics for both written and oral communication. Overall, 
students must be able to both speak and write well. 

Stating the need for quality communication seems exceedingly simple, but these 
frameworks also stated that students must be skilled in multiple forms of communication. The 
QAA-FHEQ framework stated that students must “effectively communicate infonnation, 
arguments and analysis in a variety of forms” (QAA, 2008, p. 17). Additionally, the ATC21S 
included the ability to communicate, in written or oral form, and comprehend various meanings 
in diverse situations and for different purposes (Binkley et al., 2009). 
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This outcome also pointed to the need to consider the recipient of communications. One 
Bologna competency emphasized the “ability to communicate with non-experts of one’s field” 
(Lokhoff et ah, 2010, p. 63), while several CAS standards pointed to the ability to influence 
others. Tailoring a message for a specific audience is frequently referenced implicitly, through 
mention of persuasion (via writing, speaking, and other fonns of expression), and explicitly, 
through the mention of expert and nonexpert audiences. 

A final consideration worth mentioning, although not a central facet of communication, is 
the adoption of a second language. Four of the seven frameworks—ATC21S, LEAP, Bologna, 
and the Lumina DQP—referenced the use of other languages. Indeed, although this is not 
universally agreed upon as a fundamental outcome of higher education, its repeated mention 
speaks to the expanding nature of our definition of communication. 

Digital and Information Literacy 

Over the past several decades, infonnation literacy has become an increasingly vital skill. 
Through the Internet, students now have access to an incredible wealth of infonnation that varies 
in reliability, accuracy, and intent. Thus, the ability to effectively gather, manage, and evaluate 
that information becomes increasingly vital. This was represented in our review, with each of the 
seven frameworks indicating the importance of information literacy. In short, it can be defined 
through four widely accepted outcomes: 

1. Accessing and finding infonnation 

2. Analyzing and evaluating information 

3. Using and managing infonnation 

4. Applying technology effectively 

Accessing information was discussed by five of the seven frameworks reviewed 
(ATC21S, Bologna, LEAP, QAA-FHEQ, USDOL-ETA). Specific characteristics of information 
gathering included efficiency, thoroughness, and the use of multiple resources. In particular, the 
use of technology in gathering infonnation was mentioned repeatedly. 

The ability to effectively analyze and evaluate infonnation was even more thoroughly 
emphasized, mentioned in at least 15 individual learning outcomes across seven different 
frameworks. In general, the frameworks emphasized the ability to “critically review, analyze, 
synthesize, compare and interpret infonnation” (USDOL-ETA, n.d., p. 6). 
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However, simple gathering and analysis of infonnation is insufficient in the eyes of these 
models, with each of the seven frameworks stressing the need to use information in decision 
making (e.g., “uses complex information to make decisions,” CAS, Strayhorn, 2006, p. 88), 
problem solving (e.g., using information to support critical thinking, creativity, and originality, 
ATC21S, Binkley et ah, 2009), and communicating (e.g., “incorporates multiple infonnation 
resources in different media or languages in projects, papers or perfonnances,” Lumina DQP, 
Adehnan et ah, 2011, p. 13). Several frameworks also mention that, as information is used, it 
should be done so legally and ethically (ATC21S, LEAP, Lumina DQP). 

Nearly all of the frameworks included some mention of digital literacy, but they varied 
significantly in the depth with which it was discussed. In some cases, elements of technology 
were fused into other domains. Conversely, ATC21S and USDOL-ETA both extensively 
discussed the use of technology. Aside from its inclusion in other critical domains (i.e., using 
media and electronic communication, using technology to support information gathering), we 
have identified only one outcome that was consistently represented across frameworks: applying 
technology effectively. 

Citizenship 

Citizenship has long been a desired outcome for institutions of higher education. Stemler 
(2012) found that many institutions include citizenship in their mission statements. Some have 
placed the responsibility of developing citizens on colleges and universities (Chickering, 1999; 
Colby, Ehrlich, Beaumont, & Stephens, 2003), even though academe has struggled to find a 
common definition for citizenship (Curtis-Tweed, 2004; da Silva, Sanson, Smart, & 
Toumbourou, 2004). Citizenship, however, might not seem like a requisite skill for the 
workforce, certainly not on the order of communication or teamwork skills. Nevertheless, six of 
the seven frameworks mentioned citizenship to some extent. Overall, discussions of citizenship 
tended to focus on five main points: 

1. Civic knowledge 

2. Participating in civic processes 

3. Action and organization toward change 

4. Respect for others 

5. Ethics and integrity 
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The most fundamental and often mentioned of these outcomes is civic knowledge. 

ATC21S contains six different civic knowledge outcomes, including such topics as civil rights; 
the constitution; the functions of state, local, and federal governments; democracy; and current 
events. As with many of its outcomes, the Lumina DQP focused on students’ application of this 
knowledge in communicating with others in both oral and written formats. Even the USDOL- 
ETA framework suggested that some employees possess “knowledge of laws, legal codes, court 
procedures, precedents, government regulations, executive orders, agency rules, and the 
democratic political process” (USDOL-ETA, n.d., p. 14). 

The second citizenship facet focuses on participation in civic and social processes. 
Interestingly, the nature of this activity varies both within and across frameworks. ATC21S 
referred to participation in community and neighborhood activities, including successful 
integration with organizations in the public domain, volunteering, and democratic decision 
making at all stages of government (Binkley et ah, 2009). Similarly, the CAS standards referred 
to community service. However, Bologna referred only to the “ability to act with social 
responsibility and civic awareness” (Lokhoff et ah, 2010, p. 64). 

Five of the seven frameworks also emphasize the need for students to be oriented toward 
civic change. The Lumina DQP stated that a degree holder “collaborates in developing and 
implementing an approach to a civic issue, evaluates the process and, where applicable, weighs the 
result” (Adehnan et ah, 2011, p. 20). The collaborative nature of this outcome is a common theme 
in this area, mentioned by ATC21S and LEAP, which stated that students should “collaboratively 
work across and within community contexts and structures to achieve a civic aim” (Rhodes, 2010, 
Civic Engagement Value Rubric, Capstone 4, Civic Contexts/Structures, p. 1). 

Finally, six of the seven frameworks included a prosocial component that differs from 
work in a team context, thus warranting distinction from the interpersonal skills included in the 
teamwork domain. This respect for others includes treating people fairly and with respect 
(ATC21S, CAS, USDOL-ETA), respecting and acknowledging others’ rights (ATC21S, 

Bologna, CAS), appreciation of diversity in the community (ATC21S, CAS, LEAP, Lumina 
DQP), and a sense of belonging within a community (ATC21S, CAS). 
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Life Skills 

The term life skills is often used to refer to strategies employed to facilitate success, such 
as when institutions develop a life skills course to foster competencies, such as study skills and 
time management. Here, we have adapted the tenn to include several factors that were 
consistently mentioned across these seven frameworks. Generally speaking, life skills refer to 
one’s ability to organize and manage his or her work in a way that promotes success. These 
received significant mention throughout these models, with six of the seven frameworks 
considering at least some portion of the key elements we identified. These key facets of life skills 
included the following: 

• independence, self-directed learning, 

• time management, 

• goal setting, and 

• adaptation, flexibility. 

The most widely mentioned facet within this domain was the need for students to work 
independently and to direct their own learning or work environment. Subsequent behaviors 
include initiating and working independently (ATC21S, LEAP, Bologna, CAS, QAA-FHEQ, 
USDOL-ETA), the identification of educational and professional development needs (ATC21S, 
LEAP, CAS, QAA-FHEQ, USDOL-ETA), awareness and use of effective learning styles 
(ATC21S, CAS), and continued pursuit of learning (ATC21S, LEAP, Bologna, QAA-FHEQ, 
USDOL-ETA). 

Another key skill is time management, often simply and directly stated as the efficient 
use of time and management of workload (ATC21S, Binkley et ah, 2009), “ability to plan and 
manage time” (Bologna, Lokhoff et ah, 2010, p. 63), or “manages time effectively” (CAS, 
Strayhorn, 2006, p. 3). Time management was also included in the USDOL-ETA framework. 

The ability to set goals was also referenced in four of the seven models. Just as with time 
management, the specificity with which goal setting was discussed varied significantly across 
reports. ATC21S included varying goal targets (short, medium, long-term), as well as the need to 
prioritize goals. The CAS standards contained several outcomes referring to goal quality—that 
they should be clear and realistic. 

Lastly, four of the seven frameworks emphasized the need for adaptability and flexibility. 
Given the sizable emphasis on organizational skills, it is interesting to note that students also 
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need to be able to succeed when tasks diverge from preset plans. As with time management, this 
skill was often simply stated (Bologna, QAA-FHEQ). However, the ATC21S and USDOL-ETA 
frameworks expanded somewhat, claiming that students need to function in various roles, 
responsibilities, schedules, and circumstances (ATC21S, Binkley et ah, 2009); perceive 
opportunity ambiguity and changing concerns (ATC21S, Binkley et ah, 2009); and “treat 
unexpected circumstances as opportunities to learn” (USDOL-ETA, n.d., p. 4). 

A Note About Quantitative Skills 

One area that was surprisingly absent from our findings was quantitative skills. Only 
three of the seven frameworks that we reviewed—LEAP, the Eumina DQP, and USDOL-ETA— 
included explicit mention of any quantitative domain. Even when explicitly mentioned in the 
Lumina DQP, the use of quantitative skills occurred in the context of communication and critical 
thinking: 

[A degree holder] translates verbal problems into mathematical algorithms, 
constructs valid arguments using the accepted symbolic system of mathematical 
reasoning, and constructs, as appropriate to his or her major field (or another field), 
accurate calculations, estimates, risk analyses or quantitative evaluations of public 
infonnation and presents them in papers, projects or multi-media events. (Adehnan 
et ah, 2011, pp. 13-14) 

Similarly, other frameworks, such as ATC21S, alluded to quantitative skills through the 
use of data in communication or critical thinking, but did not go so far as to define or emphasize 
the quantitative skills that should be a target of higher education or necessary in the workforce. 

There are several potential explanations for this absence. The most likely hypothesis is 
that numeracy is an assumed skill for those graduating from college. Reading is also excluded 
from our critical domains, but one could easily intuit that any person graduating from college 
would be able to read. Thus, we certainly do not want to deemphasize the importance of 
quantitative literacy. Mathematics is an important part of education, as evidenced by its presence 
on nearly every assessment of achievement or potential. Nevertheless, to include quantitative 
skills among our critical domains would not have been an accurate representation of the 
frameworks that were reviewed. 
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However, comparing these core skills (e.g., literacy, numeracy) and the more complex, 
demonstrated competencies described in this paper helps articulate a key difference in a 
framework that is used to define the outcomes of higher education versus one that identifies 
skills necessary for the workforce. By definition, an outcome is a tenninal point and not an 
underlying skill or ability. Demonstrations of outcomes contain complex applications and 
interactions of basic skills, an act that, in and of itself, is sought by higher education. However, 
in identifying those who are most likely to succeed in the workforce, measuring the individual 
underlying traits may be more valuable for a variety of reasons, such as ease of assessment and, 
similarly, more direct observation. 

Existing Educational Testing Service (ETS) Assessments 

In the next section, we address our third and final goal—reviewing existing work at ETS 
and the extent to which it aligns with our critical domains (see Table 2 for a summary). Although 
a thorough review of all of the relevant research done at ETS is beyond the scope of this report, 
we hope to evaluate two key aspects of extant work. First, to what extent has ETS developed 
assessments in these areas for higher education? That is, has this domain been researched? Have 
assessments been developed? Have these assessments been used operationally? Second, we will 
briefly evaluate the extent to which, at a conceptual or theoretical level, work at ETS aligns with 
our critical domains as they are defined. 

Creativity. The most relevant work in creativity was led by Randy Bennett and Donald 
Rock (1995, 1998), who sought to expand the scope of the GRE® General Test. Based on 
previous works by Frederiksen and colleagues (Frederiksen, 1959; Frederiksen & Evans, 1974; 
Frederiksen & Ward, 1978), they developed generating explanation (GE) items, which asked 
examinees to develop as many plausible explanations for a given scenario. These items would 
then be scored by human raters according to several criteria (e.g., uniqueness, appropriateness). 
Ultimately, the GE assessment showed promising results, with substantial reliability of scores, 
low relation with GRE general scores, lower gender and racial/ethnic gaps, and incremental 
validity in predicting graduate school success. 

One difficulty in the GE assessment involves its scoring. The criteria for a valid response 
were evaluated in these studies by human scorers, though the work of Bennett and Rock sought 
to automate this process. The cost of manually scoring assessments is high, both in time and 
resources, and may be prohibitive in some contexts. Additionally, while the GE assessment 
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appears to sufficiently address the idea generation components of creativity, it fails to address the 
application or contextual understanding components that are included in this domain. 

Critical thinking. ETS’s most direct effort to measure critical thinking in higher 
education has been the ETS' s Proficiency Profile (EPP; formerly the Measure of Academic 
Proficiency and Progress [MAPP]). Used to measure general education, EPP assesses several 
specific critical thinking outcomes. According to the most recent user’s guide (ETS, 2010), the 
EPP assesses students’ ability to 

• evaluate competing causal explanations, 

• evaluate hypotheses for consistency with known facts, 

• detennine the relevance of information for evaluating an argument or conclusion, 

• detennine whether an artistic interpretation is supported by evidence contained in a 
work, 

• recognize the salient features or themes in a work of art, 

• evaluate the appropriateness of procedures for investigating a question of causation, 

• evaluate data for consistency with known facts, hypotheses, or methods, and 

• recognize flaws and inconsistencies in an argument. 

When comparing these outcomes to the four key points identified in our synthesis 
(thinking critically, solving problems, synthesize infonnation, sense-making), it is clear that EPP 
addresses students’ ability to evaluate infonnation and arguments critically, as well as making 
sense of the critical points or themes from a given piece of infonnation. Two areas identified 
here that are not addressed by EPP are solving problems and synthesizing infonnation. 

Several components of the GRE also address aspects of critical thinking, similar to the 
EPP, mirroring many of the claims above. According to the GRE website, The GRE revised 
General Test “analytical writing measure tests your critical thinking and analytical writing skills” 
(ETS, 2013, para 1). Thus, the GRE General Test may also serve as an important reference for 
assessing critical thinking. However, it should be noted that the GRE General Test differs in both 
population (graduate education) and function (prediction, rather than outcomes assessment) from 
many of the other assessments discussed here. 

Effective communication. Assessing communication can certainly take many fonns. 
Additionally, the outcomes identified here—effectively communicating multiple types of 
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messages, communicating across multiple forms, and effectively delivering messages to varying 
audiences—cover a wide range of skills and abilities. 

Indeed, ETS has developed several assessments in this domain. The EPP includes a 
multiple-choice writing skills section that assesses students’ ability to identify appropriate 
grammar usage and organization of information. 

Both EPP and Criterion ® represent part of ETS’s established history of assessment in 
writing skills and proficiency, which also includes a great deal of work with GRE and TOEFL. 
To be sure, ETS has been an industry leader in assessing communication in this way. However, 
in reviewing these frameworks, it was clear that the definition has expanded and will continue to 
do so. Students must not only be able to write, but also do so in contexts that range from social 
media to research publications. Additionally, several frameworks emphasized the importance of 
multiple fonns of communication, particularly oral, that ETS has not assessed on a broad scale in 
this context. Although such integrative tasks take place in the TOEFL assessment, this program 
deals specifically with assessing English learners. Thus, the complex issues in communication 
uncovered here, such as the ability to consider one’s audience, require greater focus in the 
improvement of existing assessments and the development of new ones. These areas provide 
several future directions to guide research and development efforts. 

Digital and information literacy. When considering the ability to access, evaluate, and 
apply infonnation, as well as to effectively apply technology, ETS’s most relevant effort in 
higher education is clearly the iSkills™ assessment. Developed in collaboration with institutions 
of higher education, as well as experts in the field, iSkills assesses the ability to 

• evaluate the usefulness and sufficiency of infonnation for a specific purpose, 

• create, generate, or adapt infonnation to express and support a point, 

• communicate infonnation to a particular audience or in a different medium, 

• define an information problem or formulate a research statement, and 

• access, summarize and integrate information from a variety of digital sources. 

In comparing these competencies to those continually described in the literature, we see a 
great deal of alignment. The iSkills assessment not only addresses the key components of 
information literacy, but also includes a technological context that was prevalent in the 
frameworks we reviewed. Although any assessment that considers technology needs to be 
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consistently maintained for modernity and relevance, this assessment seems to be well aligned 
with the aims of higher education and its constituents. 

Teamwork, citizenship, and life skills. The remaining three areas—teamwork, 
citizenship, and life skills—have not been assessed operationally by ETS. Although considerable 
research has emerged from within ETS in the teamwork (e.g., Wang, MacCann, Zhuang, Liu, & 
Roberts, 2009) and citizenship areas (e.g., Coley & Sum, 2012), as well as the life skills areas 
such as time management (e.g., Liu, Rijmen, MacCann, & Roberts, 2009), the organization has 
yet to develop such efforts to the same extent as the other domains discussed here. 

Assessment in these areas is challenged in two ways. First, there is an issue of 
complexity. In each of these cases, as well as several others, the broad domain is not inherently 
characterized by one trait, skill, or behavior. These areas often contain components of 
knowledge, interpersonal skills, and attitudes that would require a wide variety of assessments, 
which points to the second issue in this area. 

These domains are certainly less traditional than the others identified here, most of which 
weigh heavily in the cognitive ability domain upon which ETS has focused for much of its 
existence. However, the importance of and demand for noncognitive skills continue to increase, 
as represented by a host of research, policy reports, and surveys, as well as our findings here. 
These attributes require varying types of assessment that can range from self-report personality 
measures to situational judgment tests to direct observations of behavior and skill. Ultimately, no 
one assessment can sufficiently address these complex and multifaceted domains. 

The ETSt® Personal Potential Index (PPI) 

The ETSt® Personal Potential Index (PPI) was developed by ETS in order to assess a wide 
range of cognitive and noncognitive skills in potential graduate students. Here, prospective 
students are rated by faculty and other referrers on several key behaviors designed to indicate 
such areas as creativity, communication, teamwork, resilience, planning and organization, and 
ethics and integrity. Certainly, these skills and abilities relate directly to our findings, but to 
include this as an assessment of these skills may be premature. Whether the PPI is a suitable 
assessment of these attributes depends on issues of methodology (use of behavioral observation), 
audience (graduate faculty), use (admissions rather than achievement), and intended population 
(entering graduate students). Further research should explore how the PPI and its underlying 
framework could be used to assess the achievement and potential of degree recipients. 
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Comparable Efforts 

The unique perspective of this report synthesized the frameworks of learning outcomes 
and competencies relevant to higher education. However, we are aware of three key recent 
initiatives that are highly important for this work and the effort to improve the quality and 
preparedness of college graduates in general. Here, we discuss these efforts—a recent ETS 
research project using the Department of Labor’s Occupational Infonnation Network (0*NET) 
and a separate initiative conducted by the National Research Council (NRC)—and how they 
relate to, inform, and possibly support our findings here. 

ETS 0*NET Project. In 2012, members of the Center for Academic and Workforce 
Readiness and Success at ETS began analyzing data from the Department of Labor’s 0*NET 
database (Burrus, Jackson, Xi, & Steinberg, 2013). 0*NET surveys thousands of subject matter 
experts, job incumbents, and hiring managers to collect and maintain information on the 
knowledge, skills, abilities, and work styles required for over 900 occupations. The authors 
conducted both descriptive and principal component analyses of worker characteristics and 
ultimately endorsed a model with five domains: problem solving, fluid intelligence, 
communication skills, teamwork, and achievement/innovation. Table 3 demonstrates the 
alignment between the Burrus et al. (2013) domains (along with example descriptors from the 
0*NET database) and those endorsed here. 

Table 3 


Alignment Between Burrus et al. (2013) 0*NET Domains and Critical Domains 


Critical domain 

0*NET domain 

Example 0*NET worker 
requirement/characteristic 

Critical thinking 

Problem solving 

Judgment and decision making, complex 
problem solving, critical thinking 

Life skills 

Achievement/ 

innovation 

Persistence, initiative, independence 

Teamwork 

Teamwork 

Cooperation, social orientation, 
dependability 

Effective 

communication 

Communication skills 

Oral communication, written 
comprehension, speech clarity 
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In some cases, there is clear and direct alignment. Teamwork and communication are 
clearly mentioned in both frameworks. Other concepts are unique to each approach. Burrus et al. 
(2013) identified a broad, general factor of fluid intelligence. Conversely, we identified 
citizenship, which does not appear in their framework. The remaining concepts from our 
findings—critical thinking, creativity, infonnation and digital literacy, and life skills—are 
certainly apparent in the findings of Burrus et al. (2013), but not directly apparent at the level of 
their components. For example, our findings yielded a general domain of critical thinking, which 
is partially defined by the ability to problem solve, while their framework presents problem 
solving, which is partially defined by the ability to think critically. 

Overall, these findings suggest significant overlap between these two efforts, but not 
unanimity. Differences between the findings presented here and those presented by Burrus et al. 
(2013) could be attributed to several factors, such as methodology, but a likely explanation may 
be the source material. Whereas our approach considered both educational and workforce 
perspectives, with slightly more influence coming from the former, 0*NET is strictly an 
occupational source. 

National Research Council (NRC)—21st century skills. The NRC (2008, 2010, 2011, 
2012) has recently focused on the definition, assessment, and development of 21st century skills. 
These efforts mirror the current report in several ways. Namely, they seek to synthesize literature 
to provide common definitions of the key skills needed to succeed in the workforce. Also, the 
focus on 21st century skills denotes a shift from the previous, subject-focused domains to the 
competencies that promote the application of knowledge across disciplines. 

However, the NRC efforts differ from this one in several key ways. First, they focus on 
skills that span the educational spectrum. One of the primary efforts here is to address the recent 
pressure and resulting frameworks relevant to higher education, specifically. As such, several of 
the frameworks that we considered (e.g., Bologna, LEAP, Lumina DQP) were excluded from 
their review. Moreover, the skills that they outline are derived from a review of scientific 
literature and do not focus on specific outcomes or competencies, as we have done here. 

Nevertheless, their resulting domains provide a valuable comparison for those presented 
here. Table 4 compares findings of the most recent NRC report to ours and finds significant 
congruence. Here, we see a high degree of alignment between the NRC findings and the critical 
domains presented here. Only positive core self-evaluation from the NRC framework and 
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citizenship from the critical domains are not matched between the two sets of skills. Thus, 
despite differences in the focus and scope of these efforts, the two findings are largely in 
agreement. 

Table 4 


Comparison of National Research Council (NRC) Domains and Critical Domains 



NRC domains 

Critical domains 

Cognitive 

Cognitive processes and strategies 
Knowledge 

Critical thinking 

Infonnation & digital literacy, 
effective communication 


Creativity 

Creativity 

Intrapersonal 

Intellectual openness 

Work ethic/conscientiousness 

Life skills 


Positive core self-evaluation 

N/A 

Interpersonal 

Teamwork and collaboration 
Leadership 

Teamwork 


Future Directions 

In addressing the steps that should result from this work, we have identified four key 
areas that should be considered. Although each of these has direct implications for ETS, there are 
also broader considerations for educational research and assessment development as a whole: 

• First, we must consider whether the target population of entry-level, bachelor’s 
equivalent learners is appropriate. 

• Second, from a research and development standpoint, we should consider how 
assessments in previously unaddressed domains might be constructed, and whether 
this meets the needs of various constituencies. This includes identifying the complex 
interplay among knowledge, skills, ability, behavior, and attitudes, then using this 
understanding to detennine the most appropriate assessment method. 

• Third, the critical domains have not been presented in a hierarchical or sequential 
fashion, and we should consider the sequence of each domain for various populations. 

• Fourth, we should consider the need to structure these skills for various purposes and 
uses (education, workforce, policy sector). 

• Finally, it is important to detennine how assessments in these domains might be used, 
as well as the implications of those uses. 
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Consider populations and impact. Throughout this report, we have referred to higher 
education and assumed that the key population of interest was those seeking traditional, 4-year 
bachelor’s degrees. Certainly, much of the criticism and doubt around higher education has 
targeted the rising costs associated with this level of education. However, as the role and need for 
higher education has increased in recent decades, this certainly has not been exclusive to the 
bachelor’s degree. Over 40% of currently enrolled college students attend community colleges 
(Knapp, Kelly-Reid, & Ginder, 2012), and educational certificates are a rapidly growing fonn of 
postsecondary education that provide many with an established credential (Camevale, Rose, & 
Hanson, 2012). The 0*NET system of classifying occupations is based on education and 
knowledge, as well as skill and ability requirements (i.e., occupation zones). Using the zone 
system, it is clear that frameworks are a composite of general or foundational skills required of 
all workers and tailored or specific skills, knowledge, and ability required specific to the 
occupational zone or the job itself. 

Our focus was on 4-year college graduates, who we expect to demonstrate work-ready 
skills and behaviors, general education outcomes and acquired knowledge, and college major or 
degree-specific outcomes or knowledge. Indeed, according to the literature that we reviewed 
here, traditional higher education needs to seriously consider the skills their graduates need as 
well as their level of preparation for the workforce. However, this does not limit the need or 
merit of studying similar issues at other levels of postsecondary education, such as the 
associate’s degree, master’s degree, and doctoral or professional degree. 

Frameworks From Education and Work Perspectives: Portfolio Versus Stacked, Badges 
Versus Certificates? 

One observation is that the frameworks we have reviewed reflect a combination of 
education, work-ready, and civic/life outcomes. It is hard to disconnect the framework from its 
use or purpose, primarily whether it is education or work contextualized. From a traditional 
learning outcome perspective, accreditation and other institutional accountability measures 
geared toward aggregate general learning outcomes is a mature market that ETS has historically 
addressed through its portfolio of EPP, Major Field Test (MFT; a content-based, field-specific 
test) and iSkills solutions. Other outcomes tied to student success (e.g., credit-bearing course 
credit, degree attainment) are emerging as the costs of failed postsecondary experiences become 
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evident. Certainly many of the noncognitive domains identified in the NRC framework would 
inform student readiness and success. 

Recently, the emergence of open educational resources and massive open online courses 
has generated great attention from those who work in and study higher education. The ultimate 
role of these movements remains unclear, but they hold the potential to unlock a world of 
learner-driven education—where students manage their own programs of study, learn from many 
different places, and are not necessarily educated and certified by the same organization—rather 
than the traditional, institution-driven model where students attend one institution under one 
program of study in order to receive their degree. 

Practically, this suggests a greater focus on individual certification and verification of 
specific courses or learning goals. Whether third party verification of an educational degree, per 
se, becomes an emergent need and opportunity must be monitored. Although not their explicit 
intent, frameworks of learning outcomes, such as the Lumina DQP, could infonn an interesting 
approach to creating stacked educational certificates that demonstrate an array of critical 
knowledge and skill domains. Regardless of one’s field of study, the Lumina DQP provides 
descriptions for five areas of learning: broad, integrative knowledge; specialized knowledge; 
intellectual skills; applied learning; and civic learning (Adehnan et ah, 2011). ETS should 
consider the viability of an approach such as Lumina’s, as it demonstrates how core assessment 
capabilities can be individualized and integrated into a broader portfolio of credentials. 

The work-ready agenda also is evolving amid increased economic uncertainty and 
competition. This climate requires college graduates and adults to compete for high-skill and 
livable wage jobs that are in flux as technology changes the workplace. It may be that either a 
competency-based set of proficiencies or credentials, targeting those jobs that traditionally 
require a 4-year or graduate degree (0*NET Zones 4 and 5), are a viable option to help 
individuals and workers connect from a talent supply perspective. The Department of Labor and 
National Association of Manufacturing have endorsed a more laddered framework for jobs 
requiring an associate’s degree (Zone 3) that may also be extended to higher skill occupations. In 
this framework, the foundational cognitive and behavioral skills required of all jobs underlie the 
demonstration of job-specific knowledge, skills, and assessments. Less clear is what employers 
will recognize as indicators for preferential hiring, and as Carnevale and Hanson (2012) pointed 
out, even the notion of a credential versus a certificate is complicated and a cause for confusion. 
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Allocation of resources and innovative assessments. In addition to the frameworks 
identified here, a wealth of research (e.g., Barrick & Mount, 1991; Poropat, 2009; Richardson, 
Abraham, & Bond, 2012; Robbins et ah, 2004; Robbins, Oh, Le, & Button, 2009) has shown the 
critical role that noncognitive factors play in both academic and workforce success. However, 
traditional forms of assessment have focused on cognitive ability. Recent literature reviews 
(Kyllonen, 2012; NRC, 2012) have discussed innovative assessment methods and their 
applicability in these domains. Methods such as situational judgment tests, forced-choice items, 
and anchoring vignettes are designed to address many of the issues (e.g., response bias, 
misrepresentation, cultural bias) that have been attributed to the Likert-type or self-report items 
that have long been used in this area. 

In the near future, the skills that we wish to measure may not be accessible through our 
traditional means of assessment. Thus, we need to explore these innovative item types as well as 
their practical applicability in the various settings where they may be applied. Many of the 
hurdles in introducing and scaling innovative assessment center around issues of cost for item 
generation and form construct equivalence (e.g., situational judgment tests), automation in 
delivery and scoring (e.g., behavioral interviewing and skill demonstration), and nonproctored 
high stakes testing on demand (e.g., use of biometrics in web-based delivery). 

Conclusion 

Debates over higher education’s mission and societal role have waged on for several 
decades. In addition, many have proposed frameworks for what college graduates should know 
and be able to do. Here, we have synthesized works from several perspectives and hopefully 
identified several domains that can be used to further to mold curriculum, identify assessments, 
and improve the quality of higher education. Certain issues, such as the comparison and contrast 
between higher education outcomes and workforce readiness, will continue to be topics of 
conversation but may be answered by approaches such as this. Nevertheless, as we seek to 
increase the quality of college graduates, we must first better understand the targets of the 
learning that we seek to instill. 
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Appendix 

List of Acronyms 

Acronym 

Definition 

AAC&U 

ATC21S 

CAS 

DQP 

EACEA 

EPP 

ETA 

ETS 

FHEQ 

GRE 

LEAP 

MAPP 

MFT 

NAEP 

NRC 

0*NET 

QAA 

QAA-FHEQ 

Association of American Colleges and Universities 

The Assessment & Teaching of 21st Century Skills 

Council for the Advancement of Standards in Higher Education 

Degree Qualifications Profile 

Education, Audiovisual, and Culture Executive Agency 

ETS Proficiency Profile 

Employment and Training Administration 

Educational Testing Service 

Framework for Higher Education Qualifications 

Graduate Record Examinations ® 

Liberal Education and America’s Promise 

Measure of Academic Proficiency and Progress 

Major Field Test 

National Assessment of Educational Progress 

National Research Council 

Occupational Information Network 

Quality Assurance Agency 

Quality Assurance Agency for Higher Education—Framework for Higher 
Education Qualifications 

PISA 

PPI 

USDOL 

USDOL-ETA 

VS A 

Program for International Student Assessment 

Personal Potential Index 

The U.S. Department of Labor 

The U.S. Department of Labor, Employment and Training Administration 
Voluntary System of Accountability 
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